perf(client): read prime-rl's packed completion_logprobs - #174

…2778)

`SamplingMask.from_sampling_mask` now also accepts prime-rl's packed CSR
form.

prime-rl's `/inference/v1/generate` server now returns sampling masks as
`{"ids": {data, shape, dtype}, "counts": {data, shape, dtype}}` (base64
int32 arrays) instead of `list[list[int]]`. The one-list-per-token form
was a large part of the vLLM API server's GC load and JSON size under RL
traffic (see the prime-rl PR). Here it also skips building one Python
list per token when the client parses the response.

- dict input: decode `ids` and `counts` straight into the existing int32
arrays (`_decode_packed`, which #2756 reuses for its packed logprobs).
- list input (stock vLLM): unchanged.

Companions: renderers
PrimeIntellect-ai/renderers#174 / prime-rl
PrimeIntellect-ai/prime-rl#3900. Land renderers
-> verifiers -> prime-rl; the prime-rl submodule pins must move to the
merged commits.

## Validation
- `tests/v1/test_trace.py tests/v1/test_graph.py`: pass (one new test:
packed and list inputs give the same arrays).
- `ruff check` / `ruff format` (0.16.0) on the changed files: clean.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Additive parsing path with existing list behavior preserved; main
caveat is packed `shape` is not validated during decode.
> 
> **Overview**
> Extends **`SamplingMask.from_sampling_mask`** so verifiers can ingest
prime-rl’s packed CSR sampling masks from `/generate`, not only vLLM’s
per-token `list[list[int]]` form. When the input is a dict with
**`ids`** and **`counts`**, each value is decoded via a new
**`_decode_packed`** helper (base64 `data` + `dtype` → `numpy` int32
arrays) instead of concatenating Python lists per token.
> 
> List-based input is unchanged. A unit test asserts packed and list
representations yield the same **`ids`** and **`counts`** arrays.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
8f1a6ce. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Accept packed CSR sampling masks in
`SamplingMask.from_sampling_mask`
> - `SamplingMask.from_sampling_mask` in
[types.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2778/files#diff-88a4e4ee0c58ac5dc7344161a011374082a3767eda2ad32af97c942dca982112)
now accepts either the existing list-of-lists input or a dict with
packed `ids` and `counts` arrays, matching prime-rl's `/generate` format
> - Adds a `_decode_packed` helper that base64-decodes the packed
`data`, interprets it with the packed dtype, and casts to the requested
dtype
> - Adds a test in
[test_trace.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2778/files#diff-ad6d1dda2b40e73baa4de5ad46c7a5a5580402aef93562589170bfc1bf9d1a99)
verifying that list and packed inputs produce identical `ids` and
`counts` arrays
> - Risk: the packed `shape` metadata in `_decode_packed` is ignored;
the arrays are reconstructed only from `counts` and `ids`
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
8f1a6ce.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- Macroscope's pull request summary ends here -->

…r GC (#3900)

## Problem
Under RL load (GLM-4.5-Air, 1 trainer + 1 inference node, ~130-300
concurrent 16k-token requests), the vLLM API server spends 25% (top_p
1.0) to 42-43% (top_p 0.97 + sampling replay) of its wall time in gen-2
GC, with single pauses up to 64 s. During a pause the server answers
nothing (/health, /metrics, completions, weight-update calls). With
sampling replay this cost -30% effective generation throughput and +50%
step time, and with default router health checks 25-83% of rollouts
failed.

Cause: tens of millions of GC-tracked per-token objects alive at once in
the API-server process: `Logprob` objects and dicts per token plus
decoded strings (output processor), one pydantic object per token in
`_create_tokens_logprobs`, one Python list per token for the sampling
mask, and the JSON encoding of all of it. Every gen-2 pass walks them
all.

## Change
`/inference/v1/generate` (prime-rl's `PrimeRlServingTokens`) now builds
no per-token objects:
- **Packed payloads**, encoded like `routed_experts` (one `encode_array`
helper for all three; `routed_experts.py` is removed). `routed_experts`
adds `start`, anchored at the last forwarded token: `len(prompt) +
len(completion) - 1 - len(rows)`. Without P/D this equals
`routed_experts_prompt_start`; on a P/D decode instance (#3830) the rows
begin at its first forward.
- `choice.completion_logprobs = {data, shape, dtype}`: base64 float32,
the sampled token's logprob per completion token (clamped at -9999 like
upstream). `choice.logprobs` is no longer set, so `/generate` no longer
returns `top_logprobs` (whatever `sampling_params.logprobs` asks for).
renderers asks for `logprobs = 1` and only reads the sampled logprob, so
nothing in our stack uses them.
- `choice.sampling_mask = {"ids": {...}, "counts": {...}}`: CSR base64
int32 arrays instead of `list[list[int]]`.
- **Flat logprobs, no detokenization**: non-streaming `/generate`
requests get `flat_logprobs = True` (vLLM's `FlatLogprobs`: a few lists
per request instead of a dict + `Logprob` objects per token) unless they
ask for prompt logprobs, and `detokenize = False` unless they set stop
strings (`/generate` returns token ids only, so the text and the
per-token decoded logprob strings were never used).
- **Masks stay numpy in the output processor**
(`monkey_patch_packed_sampling_masks`, non-streaming requests only):
keep only the per-step id arrays (untracked by GC) instead of a tuple
per token, and hand the serving layer a `PackedSamplingMask(ids,
counts)` instead of `list[list[int]]`.
- **API-server GC**: after app init, `gc.collect(); gc.freeze()`
(startup objects leave GC) and `gc.set_threshold(50_000, 20, 100)`
(default `(700, 10, 10)`).
- Streaming `/generate` and the OpenAI endpoints are unchanged.

Clients: PrimeIntellect-ai/renderers#174 reads
`completion_logprobs` (same checks as the per-token path);
PrimeIntellect-ai/verifiers#2778 decodes the
packed mask. Both keep the old forms for stock vLLM.

## Pins
- `deps/renderers` 41f9290 -> ca029b5 (renderers main, the merge commit
of renderers#174).
- `deps/verifiers` 2cb0a6d -> e888a71 (verifiers main, the merge commit
of verifiers#2778).

Landing order: renderers#174 -> verifiers#2778 -> this PR, with the
prime-rl re-pins to renderers/verifiers main in the same commit as the
server change. An old client against a new server gets no
`logprobs.content`.

## Measurement
One node, GLM-4.5-Air TP8, 130 concurrent closed-loop requests (Polaris
prompts, top_p 0.97 / top_k 512, max 16k tokens, ~12.5k
tokens/response), 15 min per arm, before (main) vs after (this branch):

| arm | gen tok/s | done reqs | errors | gen2 GC % wall | all GC % wall
| gen2 count | max GC pause s | health p50/p99/max s | resp MB mean |
client parse ms mean | tokens/resp |

|:------------------|------------:|------------:|---------:|-----------------:|----------------:|-------------:|-----------------:|:-----------------------|---------------:|-----------------------:|--------------:|
| base-nomask | 5497 | 512 | 0 | 11.8 | 13 | 164 | 1.87 |
0.003/1.34/21.7 | 1.71 | 28.5 | 12490 |
| fix-nomask | 5601 | 516 | 0 | 0 | 0 | 0 | 0 | 0.003/0.01/0.3 | 0.12 |
1 | 12619 |
| base-mask | 5453 | 508 | 0 | 23.8 | 25.8 | 111 | 4.27 |
0.003/4.37/30.1 | 1.83 | 31.4 | 12479 |
| fix-mask | 5446 | 504 | 0 | 0 | 0.9 | 0 | 0.99 | 0.003/0.07/1.8 | 0.31
| 1.4 | 12582 |
| fix-mask-nogctune | 5480 | 512 | 0 | 5.1 | 5.7 | 31 | 2.63 |
0.003/1.94/2.8 | 0.31 | 1.3 | 12500 |


The closed-loop tok/s does not move, because the engine core keeps
generating during API-server pauses. The RL cost came from what the
pauses block (health checks, dispatch, responses, weight updates),
measured here by /health latency. Ablation: packing alone takes gen-2 GC
from 23.8% to 5.1% of wall time; gc.freeze + thresholds take it to zero
gen-2 passes.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes the `/generate` response contract and patches vLLM internals;
clients must understand packed fields, though streaming and OpenAI
routes are unchanged.
> 
> **Overview**
> Under heavy RL concurrency, `/inference/v1/generate` no longer builds
millions of per-token Python objects in the API server.
**Non-streaming** responses now return **packed, base64 numpy payloads**
for `routed_experts`, new `completion_logprobs` (sampled-token logprob
per position), and CSR `sampling_mask` (`ids` + `counts`), with
serialization consolidated in `serving_tokens` and `routed_experts.py`
removed.
> 
> The handler enables **flat logprobs** and disables **detokenization**
when safe (no prompt logprobs / no stop strings), so upstream stops
materializing `Logprob` dicts and decoded strings. A vLLM
**output-processor monkey patch** keeps sampling-mask chunks as numpy
until finish for `FINAL_ONLY` requests.
> 
> After app startup, the API server runs **`gc.collect()` /
`gc.freeze()`** and raises GC thresholds to cut gen-2 pauses. Unit tests
cover packing and `post_process`; **`renderers`** is pinned to a release
that reads the new fields (paired client PRs).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a6f0da5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->